feat(c/sedona-extension): Add FFI table provider, exec plan, and synchronous stream exchange by paleolimbot · Pull Request #1004 · apache/sedona-db

paleolimbot · 2026-06-24T22:22:21Z

This PR adds an FFI wrapper around a TableProvider and enough infrastructure to replace an actual use case that we can test, which is importing a DataFrame from a separate SedonaDB context.

import pandas as pd
import sedona.db

sd_producer = sedona.db.connect()
sd_consumer = sedona.db.connect()

path = "submodules/geoarrow-data/ns-water/files/ns-water_water-point.parquet"

producer_sql = 'SELECT "OBJECTID", "FEAT_CODE" FROM water_point WHERE "OBJECTID" BETWEEN 10 AND 100 ORDER BY "OBJECTID"'
consumer_sql = 'SELECT "FEAT_CODE", COUNT(*) as cnt FROM df_producer GROUP BY "FEAT_CODE" HAVING COUNT(*) > 1 ORDER BY cnt DESC'

sd_producer.read_parquet(path).to_view("water_point")
df_producer = sd_producer.sql(producer_sql)

# Run the query on the producer (no FFI)
df_producer.to_view("df_producer")
result_no_ffi = sd_producer.sql(consumer_sql).to_pandas()

# Run the query on the consumer (separated via FFI)
sd_consumer.create_data_frame(df_producer).to_view("df_producer")
df_over_ffi = sd_consumer.sql(consumer_sql)
result_over_ffi =df_over_ffi.to_pandas()

# Same results!
pd.testing.assert_frame_equal(result_no_ffi, result_over_ffi)

# ...except our ImportedSedonaCExec wraps the inital plan. Projection pushed down!
explain_pd = df_over_ffi.explain().to_pandas()
print(explain_pd[explain_pd["plan_type"] == "physical_plan"]["plan"].iloc[0])
# SortPreservingMergeExec: [cnt@1 DESC]
#   SortExec: expr=[cnt@1 DESC], preserve_partitioning=[true]
#     ProjectionExec: expr=[FEAT_CODE@0 as FEAT_CODE, count(Int64(1))@1 as cnt]
#       FilterExec: count(Int64(1))@1 > 1
#         AggregateExec: mode=FinalPartitioned, gby=[FEAT_CODE@0 as FEAT_CODE], aggr=[count(Int64(1))]
#           RepartitionExec: partitioning=Hash([FEAT_CODE@0], 12), input_partitions=12
#             AggregateExec: mode=Partial, gby=[FEAT_CODE@0 as FEAT_CODE], aggr=[count(Int64(1))]
#               RepartitionExec: partitioning=RoundRobinBatch(12), input_partitions=1
#                 CooperativeExec
#                   ImportedSedonaCExec: ProjectionExec: expr=[FEAT_CODE@1 as FEAT_CODE]

This reverts commit 56f2ed6.

Copilot

Pull request overview

This PR introduces a new Sedona C-extension FFI layer for exporting/importing TableProvider and ExecutionPlan objects across process/runtime boundaries (Python/R/ADBC), enabling cross-SedonaDB-context DataFrame round-tripping while preserving DataFusion optimization opportunities (e.g., projection pushdown).

Changes:

Add a new sedona-extension crate implementing ABI-stable C interfaces for table providers, execution plans, expressions, and Arrow stream utilities.
Replace the previous Rust/Python stream-reader utilities with a shared StreamingRecordBatchReader implementation (with optional cancellation / empty-batch skipping).
Update Python/R/ADBC integrations to export/import providers via __sedonadb_table_provider__ and add Python FFI round-trip tests.

Reviewed changes

Copilot reviewed 23 out of 24 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
rust/sedona/src/reader.rs	Removes the old `SedonaStreamReader` implementation.
rust/sedona/src/lib.rs	Stops exporting the removed `reader` module.
rust/sedona/src/dataframe.rs	Adds a Sedona DataFrame extension trait + write options (currently duplicates `context.rs`).
rust/sedona-adbc/src/statement.rs	Switches ADBC streaming export to `StreamingRecordBatchReader`.
rust/sedona-adbc/Cargo.toml	Adds `sedona-extension` dependency.
r/sedonadb/src/rust/src/dataframe.rs	Switches R streaming export to `StreamingRecordBatchReader`.
r/sedonadb/src/rust/Cargo.toml	Adds `sedona-extension` dependency.
python/sedonadb/tests/test_dataframe.py	Updates comment to new `__sedonadb_table_provider__` interface name.
python/sedonadb/tests/test_dataframe_ffi.py	Adds end-to-end Python tests validating FFI DataFrame round-trips.
python/sedonadb/src/runtime.rs	Removes now-unused Rust-side “wait for future” helper.
python/sedonadb/src/reader.rs	Replaces Python stream reader with `new_py_streaming_reader` using cancellation-aware `StreamingRecordBatchReader`.
python/sedonadb/src/import_from.rs	Switches provider import to new Sedona FFI table provider capsule type.
python/sedonadb/src/dataframe.rs	Exports `__sedonadb_table_provider__` via `sedona-extension`’s exported provider implementation.
python/sedonadb/python/sedonadb/dataframe.py	Renames Python-side provider hook to `__sedonadb_table_provider__` and updates docs/comments.
python/sedonadb/Cargo.toml	Adds `sedona-extension` dependency.
Cargo.lock	Records new crate dependencies for `sedona-extension` usage.
c/sedona-extension/src/utils.rs	Adds Arrow stream/property utilities + `StreamingRecordBatchReader` implementation and tests.
c/sedona-extension/src/table_provider.rs	Adds exported/imported `TableProvider` FFI wrappers and tests.
c/sedona-extension/src/execution_plan.rs	Adds exported/imported `ExecutionPlan` FFI wrappers and tests.
c/sedona-extension/src/expr.rs	Adds exported/imported expression FFI wrappers and tests.
c/sedona-extension/src/sedona_extension.h	Extends the C header with new ABI structs and callbacks.
c/sedona-extension/src/lib.rs	Exposes new `execution_plan`, `expr`, `table_provider`, and `utils` modules.
c/sedona-extension/src/extension.rs	Adds Rust FFI struct definitions for new ABI types (error/expr/plan/provider).
c/sedona-extension/Cargo.toml	Adds dependencies needed for new FFI/provider/plan implementations.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull request overview

Copilot reviewed 29 out of 30 changed files in this pull request and generated 1 comment.

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

james-willis · 2026-07-07T23:05:31Z

+                       struct SedonaCError* err);
+
+  /// \brief Reserved for future use (must be NULL).
+  void* reserved;


these are advanced tactics heretofore unknown to me

Evolving an ABI safely without a crash as the failure mode is hard. We might want to do something else (just a brand new struct with a different name), but we can't go back and add a member to the struct and a pointer gives the most future options.

Should we be versioning the interface like a v1 suffix on all the class names?

(This is not something I'm very experienced with)

Probably we just add V2 if we ever need to update the definition (and we can mark as experimental the first time it is released, solidifying it on the second release to give us time to figure out if needs to be changed).

james-willis · 2026-07-09T05:04:13Z

 as_sedonadb_dataframe.datafusion_table_provider <- function(
  x,
  ...,
  schema = NULL,
  ctx = NULL
 ) {
  if (is.null(ctx)) {
    ctx <- ctx()
  }

  df <- ctx$data_frame_from_table_provider(x)
  new_sedonadb_dataframe(ctx, df)
 }


Does this need to be deleted?

Yes...there are no DataFusion implementations in R that produce this (we were the only one, and I removed the code that produced it).

james-willis · 2026-07-09T05:08:12Z

+
+        // Check for cancellation before fetching the next batch
+        // (periodic checking during fetch is handled by fetch_next_batch if configured)
+        if let Some(ref checker) = self.cancel_checker {


should we only check this if the periodic_check_interval is up?

james-willis · 2026-07-09T05:10:09Z

+        }
+    }
+
+    fn scan(


From our friend claude:

ImportedTableProvider::scan blocks a consumer async worker. The FFI scan call synchronously does thread::spawn + join around the producer's (potentially expensive, e.g. view planning) async scan. Consider spawn_blocking on the consumer side eventually. Also, the join()'s map_err discards the panic payload — the message is recoverable from the Box and would help debugging.

james-willis · 2026-07-09T05:17:50Z

+            ];
+            assert_batches_eq!(expected, &batches);
+
+            // Execute partition 1 - needs a fresh import since we consumed the first


is this true? looks like we reuse the same one

james-willis

couple potential issues. not sure how important they are.

james-willis · 2026-07-09T05:21:15Z

+                       struct SedonaCExecutionPlanArgs* args,
+                       struct SedonaCExecutionPlan* out, struct SedonaCError* err);
+
+  /// \brief Resolve a synchronous stream for one partition from this plan


do we need to document that the implementation should be thread safe

james-willis · 2026-07-09T06:07:31Z

+        }
+    }
+
+    fn execute(&self, partition: usize) -> Result<SendableRecordBatchStream> {


Since this is not in a tokio runtime, operators that call tokio::spawn inside execute() will panic with "must be called from within a Tokio runtime."

james-willis · 2026-07-09T06:08:45Z

+                    // This allows us to abort the fetch if cancellation is requested.
+                    let cancel_check = async {
+                        loop {
+                            tokio::time::sleep(std::time::Duration::from_millis(100)).await;


Worker-side cancellation is dead on current-thread runtimes. In streaming.rs, the worker's cancel-poll uses tokio::time::sleep inside runtime.handle().block_on. Handle::block_on on a current_thread runtime does not drive the time driver unless Runtime::block_on is running on another thread, so that timer never fires — the worker stays blocked in stream.next() indefinitely on a stalled stream (thread leak; Drop only sets the flag, never unblocks it). The reader-side recv_timeout path masks this (and is why the tests pass — note the tests themselves dodge this with std::thread::sleep and a comment acknowledging the problem). Suggest documenting that producers should supply multi-thread runtimes, or restructuring the cancel-poll to not depend on the runtime's timer.

github-actions Bot requested a review from prantogg June 24, 2026 22:22

paleolimbot changed the title ~~feat(c/sedona-extensio): Add FFI table provider, exec plan, and synchronous stream exchange~~ feat(c/sedona-extension): Add FFI table provider, exec plan, and synchronous stream exchange Jun 25, 2026

paleolimbot added 14 commits June 26, 2026 16:56

sketch structure

c76091d

sketch

aacba44

rust translation

6c0037b

error stuff

57906b0

try the expr piece

90c2fd7

move dataframe stuff to dataframe

56f2ed6

formatting

bc7d9c5

in decent shape

b5eb184

more cases

205a804

a few more spots

aa62935

add a lower-level test for the tabler provider

878c799

don't require a specific runtime

ff433c7

add a test with cancels

2a8db88

fmt

1fdef0c

paleolimbot force-pushed the ffi-table-provider branch from a7ef707 to 1fdef0c Compare June 26, 2026 21:56

paleolimbot added 9 commits June 26, 2026 17:11

use enumerate in tests

9b567a3

Revert "move dataframe stuff to dataframe"

83b0025

This reverts commit 56f2ed6.

flip the switch on Python

bb6b1f9

add non-trivial test

593af35

add the importedcexec to the explain

9aa06bd

migrate the other reader

47166d6

replace the adbc reader

7a43fba

replace Python reader

d02afad

fmt

47ad3a7

paleolimbot requested a review from Copilot June 29, 2026 03:34

Copilot started reviewing on behalf of paleolimbot June 29, 2026 03:34 View session

Copilot AI reviewed Jun 29, 2026

View reviewed changes

Comment thread c/sedona-extension/src/sedona_extension.h

Comment thread python/sedonadb/src/import_from.rs

Comment thread c/sedona-extension/src/utils.rs Outdated

Comment thread rust/sedona/src/dataframe.rs Outdated

Comment thread c/sedona-extension/src/utils.rs Outdated

address easier comments

b8d9620

paleolimbot added 11 commits July 2, 2026 16:42

fix outdated

6702794

shared cancel error constructor

1281fe3

deduplicate table provider tests

53b377a

allow get_schema to return an error

c5f006a

fix the expr signature

80b87fe

error setter macro

1c42831

fix session state export

4c65ff5

r impl

3b1fcdf

one last go

d977e0f

use an Arc<Runtime> for export/import of streams

9e08c11

maybe better cancellation

ceaca40

paleolimbot marked this pull request as draft July 3, 2026 04:15

paleolimbot added 2 commits July 6, 2026 13:40

tweak exec issues

e6d2964

pipe cancelation through both sides of the interaction

e35e663

paleolimbot requested a review from Copilot July 6, 2026 19:12

Copilot started reviewing on behalf of paleolimbot July 6, 2026 19:13 View session

Copilot AI reviewed Jul 6, 2026

View reviewed changes

Comment thread c/sedona-extension/src/streaming.rs

paleolimbot and others added 2 commits July 6, 2026 15:59

Potential fix for pull request finding

5f44619

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

periodic cancelation on the import side

e2d3a7a

paleolimbot marked this pull request as ready for review July 6, 2026 21:36

james-willis reviewed Jul 7, 2026

View reviewed changes

Merge remote-tracking branch 'upstream/main' into ffi-table-provider

88f07f7

james-willis reviewed Jul 9, 2026

View reviewed changes

james-willis approved these changes Jul 9, 2026

View reviewed changes

paleolimbot added 3 commits July 9, 2026 12:12

fix cancel checker call frequency

7438f13

don't discard panic error

3244dec

docs for safety

e4e83a2

Uh oh!

Conversation

paleolimbot commented Jun 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

james-willis left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

paleolimbot commented Jun 24, 2026 •

edited

Loading